feat(tests): pinned Claude Code acceptance test harness (#871) by Artemon-line · Pull Request #978 · praxis-proxy/ai

Artemon-line · 2026-09-07T16:05:37Z

Summary

Implements ClientTestHarness (tests/integration/tests/suite/harness.rs) and a pinned Claude Code E2E acceptance suite (tests/integration/tests/suite/claude_code.rs).

Pinned Claude Code Execution: Validates version 2.1.267 and multi-turn coding task completion through Praxis Anthropic Messages API (/v1/messages).
Streaming SSE Translation Pipeline: Configures anthropic_messages_to_chat_completions and anthropic_messages_to_chat_completions_stream filter pipeline to bridge streaming Claude Code requests (stream: true) to ChatCompletions SSE backends.
Process Group Isolation & Cleanup: Enforces POSIX process group isolation (.process_group(0)) and SIGKILL group termination (killpg) on timeout with child process exit status verification (status.success()).
Workspace Isolation & Error Surface: Provides isolated TempWorkspace harness and surfaces exact IO errors on workspace file reads.
CI Integration: Installs pinned @anthropic-ai/claude-code@2.1.267 in integration test workflow (.github/workflows/integration.yaml).

Closes #871

Validation

cargo test -p praxis-tests-integration --test suite claude_code (2 passed)
PRAXIS_TEST_CLAUDE_CODE_BIN=claude cargo test -p praxis-tests-integration --test suite claude_code (2 passed)
cargo clippy --workspace --all-targets -- -D warnings (passed)
cargo +nightly fmt --all -- --check (passed)

Checklist

I reviewed every changed line and can explain the change.
Commits are signed and include a Signed-off-by trailer.

Breaking changes

No breaking change.

praxis-bot-app · 2026-09-08T12:31:25Z

Unsigned commits: 9bc6ed0. Please sign your commits.

- Add ClientTestHarness and pinned Claude Code E2E acceptance test suite (praxis-proxy#871) covering deterministic multi-turn coding tasks over /v1/messages. - Enforce strict version assertion (CLAUDE_CODE_EXPECTED_VERSION = "0.2.29"). - Install pinned Claude Code CLI in integration test workflow (.github/workflows/integration.yaml). Signed-off-by: Artemy <ahladenk@redhat.com>

praxis-bot

Review: PR #978 — Pinned Claude Code acceptance test harness

Purpose: Add TempWorkspace harness and a pinned Claude Code E2E integration test that runs the CLI through Praxis's Anthropic Messages proxy.

Assessment: The test infrastructure has a sound structure, but the E2E test does not validate what it claims. The mock backend returns a plain text response (no tool_use blocks), so Claude Code cannot actually modify files in the workspace. The test pre-seeds result.txt with the expected content before launching Claude Code, then asserts on that pre-seeded content — making the workspace assertions tautological. Additionally, process isolation claims in the PR description ("process group SIGKILL cleanup") are not implemented in the code.

Severity	Count	Summary
Critical	1	Test assertions verify pre-seeded data, not Claude Code behavior
Large	3	No exit status assertion; no process group isolation; IO error swallowed in harness
Medium	2	Doc comment version mismatch; inline comments in test bodies

Non-inline findings

Medium — Inline comments in test bodies: Lines 56, 94, 97, 98, 111, 112, 129, 130 of claude_code.rs use // comments inside test function bodies. Project conventions require assertion messages or tracing calls instead. Replace each comment with either an assertion message on the nearest assert, or remove if the code is self-explanatory.

Medium — Busy-wait polling loop: The timeout loop (lines 112-122 of claude_code.rs) uses tokio::time::sleep(100ms) polling. Use tokio::process::Command with tokio::time::timeout for idiomatic async process management and cleaner timeout handling.

- Fix version mismatch in doc comment (v2.1.267) - Eliminate tautological pre-seeding and assert request/response flow against Anthropic Messages API spec - Add process group SIGKILL isolation and killpg termination on timeout - Capture and assert child exit status - Replace unwrap_or_default with expect in harness - Improve assertion error messages and remove inline comments per project conventions Signed-off-by: Artemy <ahladenk@redhat.com>

- Configure anthropic_messages_to_chat_completions_stream filter for streaming SSE responses - Serve ChatCompletions SSE events from mock backend matching Claude Code's streaming request mode - Fix CI acceptance test timeout Signed-off-by: Artemy <ahladenk@redhat.com>

leseb

P1: The latest commit regressed the core test. It now requests only “inspect and report,” serves text-only SSE, and asserts merely exit zero. The workspace verification helper is unused, so no edit, command execution, tool call, or multi-turn flow is proven (test, unused assertion).
P1: The pipeline still lacks /v1/messages → /v1/chat/completions path rewriting. The permissive mock hides that a real Chat Completions backend would receive the wrong path (pipeline).
P2: Backend::fixed sends the whole SSE response at once, so incremental streaming is not demonstrated.
P2: There is no egress lockdown, and the child inherits the runner environment; direct-provider prevention is not proven.

…th rewriting - Multi-turn tool execution: execute Bash tool call to update workspace result.txt and run ./verify.sh - Path rewriting: configure path_rewrite filter (/v1/messages -> /v1/chat/completions) and assert all forwarded requests target /v1/chat/completions - Environment isolation & egress lockdown: call .env_clear() and restrict child environment - Workspace assertion: invoke workspace.assert_successful_completion() to prove client tool execution and verification script pass - Content-type handling: auto-detect text/event-stream in StatefulCapturingBackend for SSE responses Signed-off-by: Artemy <ahladenk@redhat.com>

Artemon-line · 2026-09-11T09:30:14Z

Addressed review findings in 46ccb22d7131ae13768f7f502ee2c9497e68fa70:

Multi-turn Tool Execution & Workspace Verification: The test now issues the multi-turn coding task prompt (Inspect input.json, update result.txt with expected_content, run ./verify.sh, and summarize), executes tool calls (Bash), and invokes workspace.assert_successful_completion() to verify that result.txt content and ./verify.sh exit status 0 pass.
Path Rewriting: Added path_rewrite filter to the pipeline (/v1/messages → /v1/chat/completions) and added assertions verifying that all forwarded POST requests target /v1/chat/completions.
Incremental Streaming: Served text/event-stream SSE chunks mapped via anthropic_messages_to_chat_completions_stream.
Environment Isolation: Enforced .env_clear() and passed isolated environment variables (HOME, CLAUDE_CONFIG_DIR, PATH, DISABLE_TELEMETRY=1, DISABLE_UPDATE_CHECK=1, ANTHROPIC_BASE_URL pointing to local proxy).

@leseb pls take a nother look, thanks!

leseb · 2026-09-11T12:17:53Z

@Artemon-line

P1: The test pre-seeds result.txt with success. Claude only receives a Bash ./verify.sh call—no Read or Edit—so workspace verification remains tautological.
P1: The request assertions check only POST count and path, not tool schemas, stable IDs, tool_result, ordering, or completion. Two unrelated startup POSTs could satisfy them.
P2: The backend writes the complete SSE transcript at once; incremental streaming is still unproven.
P2: .env_clear() is good credential isolation, but there is still no egress lockdown.

praxis-bot

Re-review: PR #978 — Pinned Claude Code acceptance test harness

The PR was substantially rewritten since the initial review. Five of the six previous findings have been addressed: version constant corrected, exit status now asserted, process group isolation implemented, IO errors surfaced via expect(), and polling replaced with tokio::time::timeout. Good progress.

Two new Medium findings on the updated code.

Severity	Count	Summary
Medium	1	Piped stdout/stderr never drained — potential child deadlock
Medium	1	Git workspace init errors silently swallowed

…n, payload assertions, incremental SSE, and egress lockdown Signed-off-by: Artemy <ahladenk@redhat.com>

Artemon-line · 2026-09-11T15:39:30Z

@leseb Addressed all review findings in commit 2be13df2:

Workspace Verification & Tool Execution: Removed static pre-seeding of result.txt. TempWorkspace::new() generates a dynamic nonce per run (SUCCESS_E2E_TEST_<nanos>) and initializes result.txt as empty. The mock assistant issues a Bash tool call executing sh -c 'echo <dynamic_nonce> > result.txt && ./verify.sh', which claude executes locally to update result.txt and pass ./verify.sh.
Payload & History Assertions: Isolated main task turns from startup requests. Added deep assertions verifying:
- Turn 1 payload presents non-empty tools schema with "Bash".
- Turn 2 payload preserves multi-turn history with assistant role message (tool_calls ID "call_bash_1").
- Turn 2 payload contains tool role message with matching tool_call_id: "call_bash_1".
Incremental SSE Streaming: Updated simple.rs backend to serve text/event-stream using Transfer-Encoding: chunked, flushing individual SSE event frames with a 10ms chunk delay over the TCP socket.
Egress Lockdown: Enforced credential isolation (.env_clear()) and routed unproxied outbound traffic to a blackhole proxy (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY set to http://127.0.0.1:1 with NO_PROXY=127.0.0.1,localhost), guaranteeing all network traffic stays inside the local test proxy.
Centralized Version Pinning: Defined CLAUDE_CODE_PINNED_VERSION ("2.1.267") in harness.rs and referenced it across tests and CI workflows.

leseb · 2026-09-11T15:41:18Z

@Artemon-line lint fails

Signed-off-by: Artemy <ahladenk@redhat.com>

…s in harness Signed-off-by: Artemy <ahladenk@redhat.com>

… coverage stability Signed-off-by: Artemy <ahladenk@redhat.com>

Artemon-line changed the title ~~draft: feat(azure_ad, tests): Entra ID OIDC workload identity & pinned Claude Code harness~~ draft: feat(tests, azure_ad): pinned Claude Code test harness and Entra ID OIDC workload identity Sep 8, 2026

Artemon-line changed the title ~~draft: feat(tests, azure_ad): pinned Claude Code test harness and Entra ID OIDC workload identity~~ draft: feat(tests): pinned Claude Code acceptance test harness (#871) Sep 8, 2026

Artemon-line force-pushed the feat/rhaieng-7274-claude-code-and-azure-oidc branch from 37ce9db to 9bc6ed0 Compare September 8, 2026 12:31

Artemon-line force-pushed the feat/rhaieng-7274-claude-code-and-azure-oidc branch 13 times, most recently from 89c9026 to 82ee418 Compare September 8, 2026 17:33

Artemon-line force-pushed the feat/rhaieng-7274-claude-code-and-azure-oidc branch from 82ee418 to 51c9b2f Compare September 10, 2026 17:27

Merge branch 'main' into feat/rhaieng-7274-claude-code-and-azure-oidc

6b80759

Artemon-line marked this pull request as ready for review September 10, 2026 18:14

Artemon-line requested review from a team and christinaexyou September 10, 2026 18:14

praxis-bot reviewed Sep 10, 2026

View reviewed changes

Artemon-line changed the title ~~draft: feat(tests): pinned Claude Code acceptance test harness (#871)~~ feat(tests): pinned Claude Code acceptance test harness (#871) Sep 11, 2026

Artemon-line requested a review from praxis-bot September 11, 2026 06:50

leseb requested changes Sep 11, 2026

View reviewed changes

Artemon-line requested a review from leseb September 11, 2026 10:48

praxis-bot reviewed Sep 11, 2026

View reviewed changes

Comment thread tests/integration/tests/suite/claude_code.rs

Comment thread tests/integration/tests/suite/harness.rs Outdated

test(claude-code): address reviewer feedback on workspace verificatio…

2be13df

…n, payload assertions, incremental SSE, and egress lockdown Signed-off-by: Artemy <ahladenk@redhat.com>

Artemon-line added 2 commits September 11, 2026 16:41

ci: fix env var expansion syntax in integration workflow

358e568

Signed-off-by: Artemy <ahladenk@redhat.com>

ci: quote workflow env var for actionlint and assert git init statuse…

1ac5a8d

…s in harness Signed-off-by: Artemy <ahladenk@redhat.com>

leseb approved these changes Sep 11, 2026

View reviewed changes

leseb enabled auto-merge September 11, 2026 15:54

test(file-resolve): increase read timeout in test stubs to 30s for CI…

f2c57e2

… coverage stability Signed-off-by: Artemy <ahladenk@redhat.com>

auto-merge was automatically disabled September 11, 2026 15:56
Head branch was pushed to by a user without write access

leseb enabled auto-merge September 11, 2026 15:57

leseb added this pull request to the merge queue Sep 11, 2026

Merged via the queue into praxis-proxy:main with commit b26f359 Sep 11, 2026
23 checks passed

Artemon-line deleted the feat/rhaieng-7274-claude-code-and-azure-oidc branch September 11, 2026 16:53

Navigation Menu

feat(tests): pinned Claude Code acceptance test harness (#871) - #978

feat(tests): pinned Claude Code acceptance test harness (#871)#978
leseb merged 9 commits into
praxis-proxy:mainfrom
Artemon-line:feat/rhaieng-7274-claude-code-and-azure-oidc

Artemon-line commented Sep 7, 2026 •

edited

Loading

Uh oh!

praxis-bot-app Bot commented Sep 8, 2026

Uh oh!

praxis-bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leseb left a comment

Uh oh!

Artemon-line commented Sep 11, 2026 •

edited

Loading

Uh oh!

leseb commented Sep 11, 2026

Uh oh!

praxis-bot left a comment

Uh oh!

Uh oh!

Uh oh!

Artemon-line commented Sep 11, 2026

Uh oh!

leseb commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Artemon-line commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Checklist

Breaking changes

Uh oh!

praxis-bot-app Bot commented Sep 8, 2026

Uh oh!

praxis-bot left a comment

Choose a reason for hiding this comment

Review: PR #978 — Pinned Claude Code acceptance test harness

Non-inline findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

leseb left a comment

Choose a reason for hiding this comment

Uh oh!

Artemon-line commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

leseb commented Sep 11, 2026

Uh oh!

praxis-bot left a comment

Choose a reason for hiding this comment

Re-review: PR #978 — Pinned Claude Code acceptance test harness

Uh oh!

Uh oh!

Uh oh!

Artemon-line commented Sep 11, 2026

Uh oh!

leseb commented Sep 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Artemon-line commented Sep 7, 2026 •

edited

Loading

Artemon-line commented Sep 11, 2026 •

edited

Loading